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FOREWORD 


The  Instructional  Strategy  Diagnostic  Profile  (ISDP)  evaluation  workshop 
was  conducted  in  support  of  Navy  Decision  Coordinating  Paper,  Education  and 
Training  Development  (NDCP-Z0108-PN) ,  under  subproject  P.30A,  Adaptive 
Experimental  Approach  to  Instructional  Design,  and  the  sponsorship  of  the 
Chief  of  Naval  Operations  (OP-99).  The  workshop  was  held  at  the  Navy  Personnel 
Research  and  Development  Center  in  August  1977,  and  was  attended  by  repre¬ 
sentatives  from  Commander  Training  Command,  Pacific;  Chief  of  Naval  Technical 
Training;  Chief  of  Naval  Education  and  Training;  Chief  of  Naval  Education  and 
Training  Support;  and  the  Training  Analysis  and  Evaluation  Group. 

The  results  of  this  effort  will  be  provided  as  feedback  to  workshop  par¬ 
ticipants  and  to  other  commands  currently  using  or  planning  to  use  the  ISDP. 

It  is  anticipated  that  a  revised  manual  will  be  issued  in  the  first  quarter  of 
FY79.  An  interim  revision,  entitled  "An  Interim  Training  Manual  for  the  In¬ 
structional  Quality  Inventory"  (NPRDC  Technical  Note  78-5  of  February  1978), 
has  already  been  issued. 


DONALD  F.  PARKER 
Commanding  Officer 
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Problem 


Navy  Instructional  Program  Development  Centers,  under  the  Chief  of 


^ Naval  Education  and  Training  Support,  are  tasked  with  analyzing,  designing, 
and  developing  a  large  portion^ of  the  Navy's  technical  training  courses. 

JTo  facilitate  accomplishment’ of  this  task,  the  Instruction  Strategies  Diag¬ 
nostic  Profile  (ISDP)  Training  Manual  was  developed . ^  The  effectiveness  of 
this 'manual  as  a  training  instrument  must  be  evaluated  in  an  operational 
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INTRODUCTION 
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Problem 


The  Navy  Instructional  Program  Development  Centers,  under  the  Chief  of 
Naval  Education  and  Training  Support,  are  tasked  with  analyzing,  designing, 
and  developing  a  large  portion  of  the  Navy's  technical  training  courses. 

To  facilitate  accomplishment  of  this  task,  the  Instructional  Strategy  Diag¬ 
nostic  Profile  (ISDP)  was  developed  by  Courseware,  Inc.  under  contract  to 
the  Navy  Personnel  Research  and  Development  Center.1  The  effectiveness  of 
this  manual  as  a  training  instrument  must  be  evaluated  in  an  operational 
setting. 

Overview  of  ISDP 


The  ISDP  is  an  analytic  tool  that  facilitates  the  evaluation  and  revision 
of  existing  instruction  and  the  design  of  new  instruction.  It  consists  of  a 
set  of  prescriptions  that  allows  the  user  to  evaluate  instruction  logically 
and  to  prescribe  revisions  to  increase  its  effectiveness. 

The  ISDP  is  designed  to  evaluate  Instructional  materials  on  two  main 
criteria:  consistency  and  adequacy.  First,  consistency  is  determined  and 
then  adequacy  is  evaluated. 

The  consistency  criterion  is  met  if  it  is  determined  that  the  instruc¬ 
tional  objectives,  test  items,  and  the  instructional  presentation  are  con¬ 
sistent.  This  is  accomplished  in  two  steps.  First,  the  instructional 
objectives  and  test  items  are  classified  on  two  dimensions:  (1)  the  per¬ 
formance,  or  task  level,  required  of  the  student  and  (2)  the  type  of  in¬ 
structional  content.  These  two  dimensions  form  a  task/content  classification 
matrix,  which  is  used  to  classify  objectives,  test  items,  and  instructional 
presentation.  This  matrix  is  illustrated  in  Figure  1.  If  an  objective  and 
its  corresponding  test  item  can  be  classified  in  the  same  cell  of  the  matrix, 
they  are  considered  to  be  consistent.  The  second  step  involves  rating  the 
consistency  between  instruction  and  objective/test  items.  The  ISDP  requires 
that  different  components  of  instructional  presentation,  called  primary  pre¬ 
sentation  forms  (PPFs) ,  be  present  for  different  combinations  of  task 
levels  and  content  type.  If  the  PPF  combination  required  for  the  task  level 
and  content  type  of  each  objective/test  item  is  present,  then  the  instruc¬ 
tion  is  consistent  with  the  objective/ test  items. 

Once  it  has  been  determined  that  instructional  materials  are  consistent, 
the  adequacy  criterion  is  assessed  by  determining  whether  or  not  the  instruc¬ 
tional  presentation  adequately  communicates  the  "to-be-learned"  information. 
During  ISDP  development,  it  was  hypothesized  that  the  following  variables 
affected  instructional  adequacy: 


Merrill,  M.  D.,  Richards,  R.  E.,  Schmidt,  R.,  &  Wood,  N.  D.  Interim 
Training  Manual  for  the  Instructional  Strategy  Diagnostic  Profile  (NPRDC 
SR  77-14).  San  Diego:  Navy  Personnel  Research  and  Development  Center, 
September  1977. 
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Figure  1.  The  task/content  matrix  for  classification 
of  objectives  and  test  items. 
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1.  Isolation  (i.e.,  is  the  relevant  information  separated  and  clearly 
identified?). 

2.  Help  (i.e.,  is  explanatory  or  mnemonic  information  provided?). 

3.  Matching  (i.e.,  are  the  examples  and  practice  items  matched?). 

Thus,  instruction  is  rated  on  these  variables  to  obtain  an  adequacy  index. 
Each  PPF  within  the  instruction  may  be  rated  as  more  or  less  adequate. 

In  the  following  paragraphs,  the  task/ content  matrix  and  the  primary 
presentation  forms  are  described  in  greater  detail. 

Task/Content  Classification  Matrix 


As  shown  in  Figure  1,  the  task  dimension  of  the  task/ content  clas¬ 
sification  matrix  is  comprised  of  several  levels,  the  broadest  of  which  are 
the  task  requirements  Use  and  Remember.  Use  is  defined  as  the  act  of  apply¬ 
ing  a  general  relationship  to  a  specific  situation  where  it  has  not  been 
previously  applied;  and  remember,  as  the  act  of  bringing  to  mind  something 
that  has  been  previously  encountered.  Thus,  a  use  item  (or  objective)  would 
require  the  student  to  respond  by  applying  a  generality  to  a  newly  encountered 
example;  that  is,  one  that  has  not  been  previously  displayed  to  the  student 
as  part  of  the  instructional  presentation.  A  roamber  test  item  (or  objective) 
would  require  the  student  to  respond  by  recognizing  or  recalling  a  generality 
or  example  that  has  been  previously  encountered.  Generality  is  defined  as  a 
statement  or  definition  or  relationship  that  can  be  applied  to  more  than  one 
specific  object  or  event;  and  example,  as  a  specific  object  or  event  or  its 
representation  that  does  or  could  exist  in  the  real  world. 

The  use  level  cannot  be  divided  into  sublevels — it  always  requires 
newly  encountered  examples.  The  reason  for  this  is  obvious — if  an  example 
has  been  previously  encountered,  the  test  item  or  objective  would  be  clas¬ 
sified  at  the  remember  level.  The  remember  level,  however,  can  be  divided 
into  sublevels — either  paraphrase  or  verbatim.  Paraphrase  means  equivalent 
in  meaning  but  expressed  in  other  words;  verbatim,  word  for  word  or  exactly 
the  same.  Thus,  a  paraphrase  generality  means  that  synonyms  have  been  sub¬ 
stituted  for  the  substantive  words  (nouns,  verbs,  and  modifiers  of  the 
original  statement);  and  a  paraphrase  example,  that  the  same  object  or  event 
is  presented  but  the  form  or  representation  used  to  exhibit  this  object  or 
event  has  been  modified.  A  verbatim  generality/example  requires  the  student 
to  recognize  or  restate  the  same  words  that  were  used  previously  to  present  ' 
the  generality/ example.  All  paraphrase  and  verbatim  generalities  and  examples 
have  been  previously  encountered  by  the  student. 

As  shown  in  Figure  1,  the  content  dimension  of  the  matrix  involves 
four  mutually  exclusive  content  categories:  facts,  concepts,  procedures,  and 
principles.  Except  for  facts,  for  which  there  are  no  generalities,  all  types 
can  be  tested  at  any  of  the  task  levels.  These  categories  are  defined  as 
fdllows: 
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1.  A  fact  is  a  one-to-one  association  of  a  symbol  or  name  and  a 
specific  object  or  event.  For  example,  the  statement  "the  symbol  for 
resistor  is  .  .  is  a  fact. 

2.  A  concept  is  a  class  of  objects,  events,  or  symbols  that  (a) 
share  critical  attributes,  (b)  can  be  referenced  by  a  name  or  symbol,  and 
(c)  have  discriminably  different  individual  members.  For  example,  "Pumps" 
are  objects  that  move  fluids,  are  named,  and  have  different  individual 
members:  jet  pumps,  centrifugal  pumps,  etc. 

3.  A  procedure  is  a  series  of  steps  required  to  produce  an  example 
of  an  outcome  class.  Each  step  may  involve  the  temporal  or  spacing  ordering 
of  specific  objects,  events,  or  symbols  or  a  branching  of  decisions,  based 
either  on  a  fact  or  the  classification  of  an  example  of  a  concept*  A  pro¬ 
cedure  is  often  characterized  as  "how  to  do  something."  For  example,  most 
disassembly,  repair,  assembly  tasks  involve  following  procedures. 

4.  A  principle  is  a  predictive  relationship  between  specific  examples 
of  a  concept,  or  among  a  set  of  related  concepts,  which  explains  why  an 
example  of  a  particular  class  is  produced  as  a  result  of  a  particular  manipula¬ 
tion.  For  example,  explanations  of  how  current,  voltage,,  and  resistance  are 
related  in  a  circuit  involve  the  use  of  electronics  principles. 

Primary  Presentation  Forms 

The  ISDP  defines  the  instructional  presentation  fora  or  display  as 
the  fundamental  unit  of  instructional  strategy.  As  indicated  previously, 
the  instructional  presentation  form  must  meet  consistency  and  adequacy  re¬ 
quirements. 

Four  primary  presentation  forms  (PPFs)  or  displays,  which  represent 
the  various  ways  that  information  can  be  presented,  have  been  defined: 

1.  Tell  via  generality  (TG),  a  display  that  presents  a  definition 
of  a  concept,  an  algorithm  that  describes  a  procedure,  or  a  proposition  that 
expresses  a  principle. 

2.  Tell  via  example  (Teg),  a  display  that  illustrates  how  a  generality 
applies  to  a  specific  example. 

3.  Question  via  generality  (QG),  a  display  that  requests  a  definition 
of  a  concept,  an  algorithm  that  describes  a  procedure,  or  a  proposition  that  - 
expresses  a  principle. 

4.  Question  via  example  (Qeg),  a  display  that  presents  an  example  and 
requires  the  student  to  respond  to  the  example  or  presents  a  name  or  generality 
and  requires  that  student  to  respond  by  providing  an  example. 


Objective 


The  primary  objective  of  this  effort  was  to  determine  the  effectiveness 
of  the  ISDP  training  manual  in  training  participants  to  diagnose  and  recommend 
revisions  in  existing  instruction.  Secondary  objectives  were  to  acquire  a 
data  base  for  making  revisions  to  the  manual  and  to  use  these  data  in  proposing 
specific  revisions. 
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METHOD 


Training  Manual  Lessons 

The  ISDP  training  manual  was  designed  either  to  stand  alone  or  to  be 
used  as  the  basis  for  a  training  workshop,  depending  on  the  sophistication 
of  the  user.  It  consists  of  seven  lessons: 

1.  Lesson  1  provides  instruction  on  the  use  of  the  Task/Content  Matrix. 

2.  Lesson  2  provides  detailed  procedures  for  rating  test-objective  con¬ 
sistency. 

3.  Lesson  3  describes  the  primary  presentation  forms  (PPFs)  and  the 
relationship  between  these  forms  and  the  task  level. 

4.  Lesson  4  provides  instruction  and  practice  on  (a)  rating  test-item- 
presentation  consistency  and  (b)  profiling  and  indexing  test-item  presentation 
consistency. 

5.  Lesson  5  provides  instruction  on  how  to  determine  adequacy  at  each 
level . 

6.  Lesson  6  discusses  justification  of  task  level. 

7.  Lesson  7  integrates  the  data  provided  in  previous  chapters  into  a 
general  procedure  for  using  the  1SDP. 

Each  lesson  was  accompanied  by  practice  items  and  a  performance  test. 
Workshop  Participants 

The  workshop  was  attended  by  representatives  (N  =*  12)  from  the  Training 
Analysis  and  Evaluation  Group,  Chief  of  Naval  Technical  Training,  Chief  of 
Naval  Education  and  Training,  Chief  of  Naval  Education  and  Training  Support, 
and  Commander  Training  Command,  Pacific.  All  were  generally  familiar  with 
Instructional  Systems  Development  (ISD)  procedures,  but  only  one  had  been 
actively  involved  in  instructional  development  and  evaluation.  Personnel 
from  NAVP ERSRANDCEN  and  Courseware,  Inc.  conducted  the  workshop. 

Workshop  Schedule 

The  workshop  schedule  followed  the  lessons  in  the  1SDP  manual.  Partici¬ 
pants  were  encouraged  to  perform  individual  work  during  specified  periods. 
After  they  studied  each  lesson  and  took  the  accompanying  performance  test 
and  surveys,  group  discussions  were  held;  prelesson  lectures  were  avoided 
to  test  whether  the  ISDP  manual  was  usable  by  itself. 
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Dependent  Measures 


Data  were  collected  on  the  following  variables: 

1.  Lesson  study  time. 

2.  Performance  on  practice  Items. 

3.  Test-objective  consistency  rating  exercise. 

4.  Primary  presentation  forms  identification  exercise 

5.  Test-presentation  consistency  exercise. 

6.  Presentation  adequacy. 

7.  Final  simulation  exercise. 

8.  Lesson  opinion  questionnaire. 


RESULTS 


Lesson  Study  Time 

The  total  contact  time  with  the  ISDP  lesson  materials,  tests,  and  feed¬ 
back  sessions  over  the  5-day  workshop  was  approximately  30  hours.  Par¬ 
ticipants  spent  an  average  of  12  hours  and  19  minutes  on  lesson  materials, 
with  a  range  from  9  hours  and  30  minutes  to  17  hours  and  50  minutes.  The 
remaining  time  (17-1/2  hours)  was  spent  in  tests  and  feedback. 

Performance  on  Lesson  Practice  Items 


Workshop  participants  were  asked  to  tally  their  correct/ incorrect  re¬ 
sponses  to  the  practice  items  in  Lessons  1  through  5.  Table  1,  which  pro¬ 
vides  the  mean  percent  of  correct  responses  achieved  by  participants,  shows 
that  they  performed  fairly  well  on  Lessons  1  through  4,  but  not  on  Lesson  5. 
This  poor  performance  indicates  that  either  the  task  or  the  instructional 
segment  for  Lesson  5  was  too  complex  or  difficult  to  understand,  or  the 
previous  instruction  did  not  adequately  prepare  the  participants  for  competent 
performance.  Since  students  cannot  successfully  complete  Lesson  5  without 
a  thorough  understanding  of  constructs  represented  in  Lessons  1  through  4, 
it  appears  that  previous  material  was  not  adequate. 


Table  1 

Proportion  of  Correct  Responses  on 
Lesson  Practice  Items 


Lesson 

Mean  Percent 
Achieved 

1. 

Task/ Content  Classification 

.677 

2. 

Test-Objective  Consistency 

.773 

3. 

Primary  Presentation  Forms 

.741 

4. 

Test-Presentation  Consistency 

.686 

5. 

Presentation  Adequacy 

.285 

Test-objective  Consistency  Rating  Exercise 

Lesson  1  taught  participants  to  classify  test  items  and  objectives  accord¬ 
ing  to  the  task/content  matrix;  and  Lesson  2,  to  determine  the  consistency 
between  test  items  and  objectives.  After  studying  these  two  lessons,  par¬ 
ticipants  were  given  a  list  of  16  objectives  and  16  test  items  and  were 
directed  to  determine  (1)  the  correspondence  between  test  items  and  objectives, 
(2)  the  task  levels  and  content  types  of  test  items,  and  (3)  the  task  levels 


and  content  types  of  objectives.  Based  on  the  proportion  of  correct  responses 
from  each  participant,  an  overall  reliability  of  .70  was  achieved,  which  was 
equivalent  to  performance  on  the  practice  items  in  Lessons  1  and  2  (Table  1). 


Primary  Presentation  Form  Identification  Exercise 


After  studying  Lesson  3  on  primary  presentation  forms  (PPFs),  partici¬ 
pants  were  given  a  three-page  excerpt  from  a  Navy  instructional  manual. 

This  excerpt  provided  ten  displays — seven  of  which  were  "Tell"  displays; 
and  three,  "Question"  displays — outlined  with  a  marking  pen.  The  task  was 
to  classify  each  of  the  displays  according  to  the  type  of  PPF:  TG,  Teg, 
Qeg,  QG,  or  "other."  Reliability  was  estimated  by  the  percent  of  correct 
classifications  on  all  displays  by  all  participants;  The  reliability  for 
all  displays  was  .75;  for  the  seven  TG/Teg  displays,  .65;  and  for  the  three 
Qeg/QG  displays,  .96.  The  difference  in  reliability  between  the  "Tell"  and 
"Question"  displays  may  be  because  the  latter  provide  an  obvious  cue  or 
hint  in  prose  text,  while  "Tell"  displays  do  not. 


Test-presentation  Consistency  Exercise 


Decisions  made  about  content  type  are  often  independent  of  the  task 
level  of  the  test  item,  while  those  made*  about  the  types  of  PPFs  necessary 
to  teach  to  a  test  item  depend  on  its  task  level.  Therefore,  if  participants 
chose  the  incorrect  test  item  task  levels,  they  would  not  arrive  at  the  cor¬ 
rect  PPF  index  (see  Figure  2). 


After  studying  Lesson  4,  participants  completed  a  test-presentation  con¬ 
sistency  exercise.  Only  53  of  240  responses  obtained  were  correct,  yielding 
a  reliability  measure  of  22.1  percent.  Participants  experienced  particular 
difficulty  in  classifying  the  task  level  of  test  items.  Moreover,  even  when 
they  chose  the  correct  task  level,  they  arrived  at  the  correct  PPF  index  for 
less  than  half  of  the  items. 


Presentation  Adequacy  Exercise 


After  studying  Lesson  5,  participants  were  given  a  segment  of  Navy  in¬ 
struction  with  five  accompanying  test  items  and  asked  to  use  a  designated 
ISDP  rating  form  to  rate  the  instruction  associated  with  each  item.  Several 
participants  either  failed  to  complete  the  rating  task  on  all  given  forms  or 
failed  to  respond  to  several  questions  on  particular  rating  forms.  Thus, 
there  were  several  "holes"  in  the  data  for  this  test.  The  overall  poor  per¬ 
formance  could  be  interpreted  in  several  ways: 


1.  The  rating  forms  are  too  difficult  to  use. 


2.  The  instruction  was  too  difficult  to  rate. 


3.  The  ISDP  lesson  instruction  had  not  provided  sufficient  practice  or 
clarification  of  instructional  adequacy  concepts. 


Also,  as  reported  above,  there  were  some  problems  with  Lesson  5  practice 
items,  as  Indicated  by  an  average  proportion  correct  of  .285  shown  in 
Table  1. 


Final  Simulation  Exercise 


An  instructional  segment  of  approximately  42  pages  taken  from  a  basic 
Navy  Electronics  course  was  used  in  a  simulation  exercise  to  determine 
whether  participants  had  acquired  the  skills  identified  in  Lessons  1  through 
5.  In  this  exercise,  participants  were  told  to  consider  that  they  were  in 
an  instructional  evaluation/ revision  job,  where  they  had  to  analyze  the  in¬ 
struction  and  write  a  summary  report  to  the  project  director  for  his  use  in 
proposing  course  improvements  to  the  commanding  officer.  Also,  they  were 
permitted  to  decide  what  sequence  they  would  follow  in  creating  a  final 
summary  product.  A  latent-image  response  booklet  was  used  to  record  each 
response  and  to  direct  students  to  the  location  of  sections  that  were  or 
were  not  relevant  to  the  analysis.  In  making  each  response,  the  student 
would  mark  the  time  required,  so  that  summary  "cost-effectiveness"  could 
be  estimated  based  on  the  time  spent  and  test  scores  obtained. 

Table  2  provides  the  amount  of  time  spent  (in  minutes)  on  the  final  test, 
along  with  corresponding  test  scores  based  on  a  rating  of  the  effectiveness 
of  instruction  analysis.  As  shown,  the  time  spent  ranged  from  78  to  141 
minutes;  and  scores,  from  23  to  71  (a  maximum  possible  score  was  78).  The 
Pearson  r  correlation  between  time  and  performance  was  .05,  indicating  that 
there  is  no  relationship  between  the  amount  of  time  spent  and  the  effective¬ 
ness  of  the  rating. 


Table  2 

Total  Time  and  Score  on  the 
Final  Simulation  Exercise 


Subject3 

Time 

(minutes) 

Test 

Score 

1 

85 

28 

2 

99 

91 

3 

97 

91 

4 

141 

37 

5 

78 

35 

6 

103 

29 

7 

100 

69 

8 

118 

54 

9 

86 

31 

-  10 

105 

41 

X  -  101.2 

X  -  50.6 

Note.  Pearson  r  correlation  between  Time  and  Score  was  .05 


*Two  subjects  did  not  complete  the  test. 
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•*>  .V*r 


Participants  required  an  average  time  of  1  hour,  41  minutes  to  complete 
this  exercise.  Thus,  if  this  lesson  is  considered  representative  of  other 
lessons  in  a  course  of  instruction,  in  terms  of  both  length  and  complexity, 
and  if  a  course  consisted  of  30  such  lessons,  it  could  be  estimated  that  the 
actual  course  would  require  approximately  50  hours.  Given  that  a  course 
would  best  be  accomplished  with  the  help  of  subject  matter  experts  working 
in  conjunction  with  an  instructional  developer/evaluator,  a  total  expenditure 
of  100  man-hours  could  produce  a  complete  diagnostic  profile  for  a  course. 

Some  participants  selected  incorrect  rating  forms  according  to  task 
levels,  which  indicates  that  Lesson  7  of  the  ISDP  manual  does  not  provide 
adequate  instruction  on  the  use  of  the  ISDP. 

However,  participants  had  a  positive  attitude  toward  the  simulation 
exercise,  as  indicated  by  their  perseverance  in  performing  the  task  and 
verbal  statements  made  to  workshop  instructors. 

Lesson  Opinion  Questionnaire 

After  studying  each  lesson,  participants  were  asked  to  rate  its  adequacy 
in  the  following  areas:  generalities,  examples,  practice,  feedback,  graphics, 
overall  design,  importance,  and  strengths  and  weaknesses.  Responses  were 
made  on  a  five-point  Likert-type  scale.  Results  showed  that  (1)  the  lesson 
examples  were  judged  to  be  more  effective  than  lesson  generalities,  and  (2) 
Lesson  5  was  judged  to  be  the  least  adequate;  and  Lesson  3,  most  adequate. 
While  there  seems  to  be  a  relationship  between  participant  opinion  and  per¬ 
formance,  it  should  be  noted  that  tasks  in  Lesson  5  were  much  more  difficult 
than  those  in  Lesson  3.  It  appears  that  opinions  about  Lesson  5  would 
improve  if  the  "richness"  of  the  instruction  matched  the  complexity  of  task 
requirements. 
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DISCUSSION  AND  CONCLUSIONS 


Training  Workshop 

1.  Workshop  Format.  The  workshop  format  for  training  in  the  use  of  the 
ISDP  is  a  necessary  adjunct  to  the  manual.  At  present,  the  manual  cannot  be 
viewed  as  completely  exportable  or  self-instructional. 

2.  Workshop  Time.  The  5-day  time  period  for  the  workshop  was  adequate. 
However,  if  the  content  of  the  manual  is  improved,  less  time  would  be  required. 

As  the  manual  stands,  mastery  of  ISDP  techniques  cannot  be  achieved  during  1 
working  week.  Additional  on-the-job  applications  must  be  made  to  enhance 
skills  acquired  in  the  workshop.  Also,  since  everyone  does  not  need  to  master 
the  skills,  perhaps  short  workshops  could  be  designed  to  provide  managers 
with  moderate  understanding  of  the  techniques,  while  longer  workshops  would 

be  appropriate  for  curriculum  designers  and  evaluators  who  would  use  the  skills 
on  a  daily  basis. 

3.  Performance  Tests.'  Additional  performance  tests  should  be  designed  to 
provide  other  measures  from  diverse  subject  matter,  content,  and  task  types. 
Additional  time  could  be  spent  on  testing  and  feedback,  since  the  feedback 
seemed  to  help  participants  in  understanding  the  techniques. 

4.  Workshop  Participants.  The  number  of  workshop  participants  should  be 
limited  to  10  or  12  people  with  fairly  similar  backgrounds  and  experience  to 
facilitate  interaction  and  to  provide  individual  help.  Several  prerequisites 
should  be  required  for  participants,  such  as  a  familiarity  with  instructional 
objectives,  tests,  and  print  delivery-system  type  (text)  instruction.  Formal 
training  in  instructional  technology  and/or  experience  in  ISD  is  necessary  for 
successful  participation. 

Simulation.  The  final  or  posttest  performance,  in  the  form  of  a  simula¬ 
tion  exercise,  was  useful  for  assessing  the  extent  to  which  integration  of 
skills  had  been  acquired.  More  than  one  posttest  simulation  should  be  provided 
to  enhance  performance.  Modularly  constructed  simulations  could  be  designed 
to  provide  subject  matter  examples  that  are  familiar  to  specific  participant 
groups,  although  this  would  take  considerable  effort. 

6.  Follow-up .  Follow-up  surveys  or  interviews  should  be  conducted  (a) 
to  determine  the  extent  to  which  ISDP  techniques  are  used,  and  (b)  to  identify 
technical  and  management  problems  that  are  typically  encountered  in  the  process. 
This  would  allow  the  assessment  of  long-range  implementation  problems  that  can¬ 
not  be  evaluated  in  relatively  short  workshops. 

ISDP  Manual 

1.  Typographical  and  Design  Errors.  Since  some  participants  were  obviously 
distracted  by  the  mechanical  errors  in  the  manual,  future  tryouts  of  the  manual 
should  be  made  with  corrected  copies. 

2.  Rating  Forms.  The  rating  forms  should  be  standard  throughout  the  manual, 
since  the  need  to  adjust  to  new  rating  forms  in  different  lessons  caused  con¬ 
siderable  confusion.  Further,  rating  forms  can  be  simplified  without  loss  of 
information.  From  the  consents  made  by  the  participants,  this  improvement  would 
have  considerable  impact  on  whether  potential  users  would  use  the  ISDP  procedures. 


3.  Terminology.  Several  participants  found  the  terminology  difficult 
to  understand  and  use.  Thus,  the  number  of  unfamiliar  terms  should  be 
minimized,  and  when  these  terms  are  introduced,  familiar  usages  should 

be  chosen. 

4.  Examples.  Example  sets  in  the  manual  should  focus  on  Navy  subject 
matter  as  much  as  possible  while  maintaining  diversity. 

5.  Paraphrase/Verbatim.  It  was  difficult  for  participants  to  distinguish 
between  paraphrase  and  verbatim  statements,  even  when  instruction  was  provided. 
Further,  the  significance  of  the  distinction  for  instructional  design  was 
questionable.  Thus,  the  paraphrase/verbatim  distinction  should  be  removed 
from  the  task  content  matrix.  Only  when  it  can  be  shown  that  different 
instructional  presentations  are  required  that  result  in  different  student 
performance  can  such  distinctions  be  supported. 

6.  Generalities.  Some  generalities  in  the  manual  (such  as  procedures 

for  completing  rating  forms)  seem  to  be  too  complex  or  present  so  much  in- 
formation  that  they  are  not  practical  even  when  cast  in  the  form  of  a  pro¬ 
cedural  algorithm.  These  generalities  would  be  more  useful  if  less  informa¬ 
tion  were  presented  and  alternative  representations  for  facilitating  encoding 
and  comprehension  were  offered.  - 

7.  Indexing.  The  indexing  procedure,  which  provided  a  numerical  estimate 
of  relative  instructional  quality,  was  not  useful  for  the  revision  process. 
Revisions  depend  upon  identifying  specific  problems  in  instruction,  which  was 
lost  in  the  indexing.  The  rating  forms  should  be  designed  to  call  for  yes/no 
responses  to  questions  about  instructional  quality,  with  each  "no"  response 
indicating  a  particular  required  revision. 

8.  General  Considerations.  The  participants  agreed  that  the  basic  assump¬ 
tions  of  the  ISDP  were  sound  and  potentially  very  useful  in  various  Navy  train¬ 
ing  applications.  They  also  agreed  that  the  manual  and  the  workshop  should  be 
improved  as  described  above,  if  the  techniques  are  to  be  widely  applied.  Some 
observers  were  concerned  about  the  amount  of  time  required  to  analyze  instruc¬ 
tion  using  the  ISDP.  However,  if  conducting  such  an  analysis  leads  to  cost- 
effective  revisions  rather  than  to  redesign  and  production  of  an  entire  course, 
the  initial  investment  of  analysis  time  could  well  be  worthwhile.  This  cost- 
effectiveness  issue  should  be  further  studied.  Also,  additional  monitored 
tryouts  of  the  technique  should  be  conducted  to  obtain  a  more  precise  estimate 
of  the  time  required. 


RECOMMENDATIONS 


It  is  recommended  that: 

1.  The  manual  be  revised  as  indicated  in  the  previous  section.2 

2.  Additional  workshops  be  given  to  test  revisions. 

3.  The  cost-effectiveness  of  the  ISDP  process  be  evaluated  by  applying 
it  to  existing  Navy  courses.  Analysis  should  be  made  of  time  required  to 
apply  the  ISDP  and  student  performance  data  on  the  revised  materials. 


2Since  the  ISDP  workshop  was  conducted  and  results  evaluated,  the  In¬ 
structional  Strategy  Diagnostic  Profile  has  been  retitled  as  the  Instructional 


Quality  Inventory  (IQI).  Also,  some  recommended  revisions  resulting  from  this 
effort  have  been  Incorporated  in  the  Interim  Training  Manual  for  the  Instruc¬ 
tional  Quality  Inventory.  NPRDC  Technical  Note  78-5  of  February  1978. 
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